The Curse of Optimism

Stuart I. Reynolds sir s.bham.a .uk S hool of Computer S ien e, The University of Birmingham, Birmingham, B15 2TT, UK 1. The max Operator and Optimisti Initial Biases This paper introdu es the following simple insight: RL algorithms that update their value estimates based upon a return estimate involving maxaQ(s; a) nd it more diÆ ult to overome their initial value biases if these biases are optimisti . To see that this is so, onsider the example in Figure 1. Assume that all transitions yield a reward of 0. A learning algorithm is applied that adjusts Q(s1; a1) towards E[maxaQ(s2; a)℄. If all Q-values are initialised optimisti ally, to 10 for example, then the Q-values of all a tions in s2 must be readjusted (i.e. lowered towards zero) before Q(s1; a1) may be lowered. However, if the Q-values are initialised pessimisti ally by the same amount (to -10), then maxaQ(s2; a) is raised when the value of a single a tion in s2 is raised. In turn, Q(s1; a1) may then also be raised. In general, it is lear that it is easier for RL algorithms employing maxaQ(s; a) in their return estimates to raise their Q-value predi tions than to lower them. In e e t, the max operator auses a resistan e to hange in value updates that an inhibit learning. It is also lear that the e e t of this is further ompounded if i) the Q-values in s2 are themselves based upon the over-optimisti values of their su essors, or, ii) states have many a tions available, and so many Q-values to adjust before maxaQ(s; a) may hange. Examples of methods that use maxaQ(s; a) in their return estimates and are a e ted by this phenomenon are: value-iteration (Sutton & Barto, 1998), Qlearning (Watkins, 1989), R-learning (S hwartz, 1993), Watkins' Q( ) (Watkins, 1989; Sutton & Barto, 1998), Peng and Williams' Q( ) (Peng &Williams, 1996) and their many derivatives. Experimental results with value-iteration and Qlearning have shown that, when the e e ts of optimism on the agent's exploration strategy are a ounted for, onvergen e towards the optimal Q-fun tion generQ(s , a ) = 10 1